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ABSTRACT 

This document praises the Comprehensive Preschool 
Education and Child Day-Care Act and advocates an equally 
comprehensive collateral program of research and evaluation. In order 
to avoid delay in starting preschool and child care programs, the 
research undertaken should be of the kind called evaluative research 
in which program and research are in progress simultaneously. The 
research model is therefore integrated into the program and focuses 
on process as well as product measures. Although traditional research 
has followed an engineering model, the research associated with these 
preschool programs should follow a medical model. The implications of 
a medical model include assessment of both intended and possible 
outcomes, and frequent monitoring of participants* feelings and 
program processes. An example of such research is the Head Start 
longitudinal study of disadvantaged children being carried on by 
Educational Testing Service. Included is a description of some of the 
problems encountered in this project. (MH) 
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Samuel Messick* 



I welcome the opportunity to comment on the '''Comprehensive Preschool 
Education and Child Day-Care Act of 1969" and will particularly es^hasize 
the need for a correspondingly comprehmisive collateral program of research 
and evaluation in connection with this effort. 

The coH^rehensiveness of the, proposed programs is a major strength of 
the Act, for it properly recognizes— by including provision for physical and 
mental health services, food and nutritional services, educational activities, 
special programs, and social services to improve hoaie environments and to en- 
sure parental involvmnent — that we are dealing in the area of human develop- 
ment with a complicated system composed of many interdependent elements. A 
child’s intellectual functioning is not independent of his personal-social 
and affective functioning nor of the state of his physical well-being. F\ir- 
thermore, the factors influencing individual growth and development are also 
multiple and overlapping, unbracing not only school, teacher, and program 
* influences but family, peer, and community influences as well. The compre- 
hensiveness of the Act affirms this interrelatedness of personal, social, 
environmental, and educational factors in the development of human potential. 

BecaAise the child-process-environment interactions addressed by the pro- 
posed program of educational, health, and social services is complex and 

cc 






little understood and ‘because we do not yet have adequate knowledge to 
develop optimal procedures for implementation, it is imperative that inten- 
sive programs of research he undertaken to increase our understanding of 
human development and of methods for enhancing it. However, this sho\ild not 
he taken to indicate, as seme have argued, that comprehensive action programs 
should he postponed until we are better prepared. There is an alternative 
strategy, and that is to undertake what has come to he called evaluative 
res earch . 

With this strategy, we would go ahead and develop the program using the 
heat available knowledge and ideas — or, better still, develop several options 
incorporating various alternative approaches — and then we would proceed to 
carry out the program and research simultaneously. This would he accomplished 
by including within the administration of the program provision for collecting 
information relevant to its evaluation and improvement. It is sometimes even 
possible to embed the evaluative research directly into the program itself by 

\ t 

capitalizing upon certain kinds of information, such as measures of education- 
al progress, for both instructional and research purposes. 

It is important, however, that the scope of the evaluative research 
* match the comprehensiveness of the program concerns and deal in a meaningful 
way with the complexity of interacting influences affecting child behavior and 
development. To do this requires a substantial change in approach from the 
traditional evaluation study of the past, which typically cast the problem as 
a kind of "horse race"; The research question asked was of the form, "Is this 
program or treatment more effective than that one or more effective than no 
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progrsro at all?" This is akin to ""brand name" or "product" testiig, where 
the major concern is with demonstrated differences in outcome. 

But if the research is to contribute to the improvement of the program, 
it must focus not only upon product but upon process, and it must ask 
different and more complicated kinds of questions. For example, answers to 
the "horse race" type of question are usually sought by comparing average 
gains in specific achievement for children receiving one treatment with 
average gains for children receiving a second treatment. But suppose treat- 
ment 1 is better for certain kinds of children (as a function, say, of their 
attitudes or interests or cognitive styles) and treatment 2 better for other 
kinds of children. Depending upon the mix of children in the two groups, the 
two treatments might exhibit negligible differences on the average ^ whereas, 

actually, they produce wildly different effects upon individuals. A completely 
different evaluation and understanding of the treatments might have resulted 

if some other question had been asked, such as "Do these treatments interact 

■\ 

with personality and intellectual characteristics of the children or with 
factors in their educational history or family background to produce different 
effects upon performance? What dimensions of educational experience are asso- 
ciated with growth on dimensions of cognitive functioning or with changes in 
attitude or affective involvement, and what social and environmental factors 
moderate the impact?" Rather than asking what program works best, we should 
be asking what procedures work best for what kinds of children under what 
kinds of circumstances. 

The traditional approach to program evaluation might be said to follow a 
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manufacturing or engineering model, where attention is focussed upon differ- 
ences in outcome or product ^ or upon input-output differences relative to cost. 
The approach advocated here ini^t he said to follow a medical model and there 

« 

are several important consequences of this conception. 

To begin with, there is the recognition that a prescription for treat- 
ment and the evaluation of its effectiveness should taJtee into accomt not only 
the reported symptoms hut other aspects of the organism and its ecology as 
well. This is essentially another affirmation of the need to deal empirically 
in evaluative research with the interrelatedness of psychological, social, 
environmental, and educational factors. 

Another derivative of the medical model is a concern for monitoring 
possible side effects of the treatment. This also follows naturally from the 
recognition that we are dealing with a complex system composed of interdepen- 
dent parts and that a change in one part of the system may produce unantici- 
pated and possibly adverse consequences in another part of the system. 

Because of this possibility, it is not enough to evaluate a program solely in 
terms of its stated goals, on the basis of how well it achieves its intended 
objectives. In addition to the intended outcomes, we should also assess a 
wide range of possible outcomes, for we might unearth in the process some 
alternatives that ought to be weighed in reaching a final appraisal of program 
impact . 

Another implication of this medical analogy is that feelings and reac- 

I 

tions of the program participants should be assessed periodically throughout 
the course of the treatment and not Just at the beginning and the end. This 



PS 003073 



-5 



• assafismsnt should include, as is the custom in medical practice, a monitoring 
of attitudes tovard the treatment itself. 

And, of course, underlying the entire metaphor is the notion that when- 
ever possible in the evaluation of ©ducational programs, as in the evaluation 
of drugs, we should go beyond a simple assessment of the size of effects to 
an investigation of the processes that produce the effects, for an understand- 
ing of these processes will provide a rational basis for improving the program, 
for changing programs if conditions change, and for isolating possible danger 
zones where potential side effects shotild be monitored. 

In this view, then, evaluative research should focus not only upon the 
outcomes of the program but also upon the process and context of the program. 

In addition, it is probably wise to extend the range of the research even 
further In time. As in a medical case study, measures of antecedent conditions 
should be included, as well as follow-up measures of the consequences both of 
the treatment and of the termination of treatment. For these purposes, the 
scope of measurement must be broad enough both to ensure adequate coverage of 
potentially interactive variables and to permit the monitoring of possible side 
effects of the treatment. If possible, measures should be included to assess 
not only characteristics of the learners but also of their learning environ- 
ments (including the hcane and community as well as the classroom and school), 
and of the educational and treatment processes at all levels (including charac- 
teristics of teachers, programs, and classroom dynamics). In this latter con- 
nection, it is particularly important to assess characteristics of the program 
as it is actually carried out, since the program as practiced is sometimes 
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quite different from the program as planned. 

A might be helpful at this point to describe an instance of this type 
eraluative research, although the example chosen is on a much more massive 
•^oala than most evaluative research efforts. It is the ’'Longitudinal Study 
of Disadvantaged Children and Their First School Experiences, an evaluative 
research study of Head Start programs being conducted by Educational Testing 
Service under the auspices of the HEW Office of Child Development and the 



Office of EconcMic Opportunity . 



Field operations for the study were initiated in the summer of 1968, after 
a year of planning and instrument development. The Head Start centers selected 
for investigation were in four localities: St. Louis, Missouri; Trenton, New 

Jersey; Portland, Oregon; and Lee County, Alabama. 

To identify subjects for the study, target elementary schools were first 
located within these four sites. Then their sending districts were determined 
and canvassed to identify every child who will be eligible for enrollment in 
first grade in the Fall of 1971 * This turned out to be a sample of close to 
1800 children. At the same time, an assessment battery was developed that 
would provide base-line information on the important dimensions of cognitive/ 
intellectual and personal/social functioning. 

The schools were chosen in areas in which a high proportion of children 
are eligible for Head Start. Since Head Start is a voluntary program, not all 
of the eligible children will participate and, in any event, the available 
Head Start facilities in these areas would very likely handle all of them. 
Hence, the sub-samples of interest will form naturally— : some children will go 
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to Head Start classes and some will not. It is anticipated that about half 
of the subjects will have Head Start experience. In the Spring of 1970, all 
of the subjects will be reassessed, as they will each succeeding year until 
they are in third grade. For those children attending Head Start classes it 
is planned to observe classroom dynamics and to assess the nature of the pro- 



grams through direct observation of actual classroom operations — not merely 
by looking at the prospectus for the program or at intentions of the teachers. 

A physical examination of the child will also be included periodice.lly. 

In addition, the family will be interviewed each year, not only to ascertain 
the usual socioeconomic status characteristics but also to assess family pro- 

I 

cess variables, such as child-rearing attitudes and practices. Family informa- 
tion will be obtained through interviews, through some tests of the parents, 
and through observation. Characteristics of the community and the school will 
also be assessed as well as characteristics of the classroom and the program, 
the latter being obtained through direct classroom observation. Teacher charac- 
teristics will be assayed through interviews and tests. 

One of the major problems that had to be faced in this study was one of 
feasibility — was it possible to go into these communities, some of which v^ere 
ghetto communities, and actually collect the data? The researchers would have 
to be welcome in these neighborhoods for a six-year period, were asking for a 
sizeable contribution of time and effort from families, and were going to try 
to set up centers to assess 3 1/2-year-old children. The logistics of this, 
as might be imagined, are extremely complicated. 

Another problem of .feasibility was whether or not it would be possible, 

. M ' ■- ; 
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in any sensible length of times "to assess the important characterisitics of 
children this young. An assessment battery was assembled which required up~ 
wards of 6 hours to administer, and there was some concern a,s to whether such 
intensive testing was possi.ble with a 3 l/2--year“Old child. As it turns’ out, 
this is not an undue burden for the child — what we view as 6 hours of testing 
is evidently seen by the child as six hours of one-to-one contact with an 
interested and attentive adult. The six hours of testing is spread over a 

I 

five-day week, 'during which the child attends the center for three hours a day, 
so there is ample opportunity for rest and for play activities. One hour of 
this testing includes a parent-child interaction task that has been very mean- 
ingful in assessing the language characteristics and teaching styles of the 
mother . 

Another serious problem that had to be faced was how to find and train 
appropriate people to do the assessment of the children. It was feared that 
even if it were possible to locate sufficient numbers of college-level person- 
nel, such as undergraduate or graduate students, who would be available to 
work full-time during a two or three month period in the Spring, it would 
probably not be feasible to send them into these communities to try and collect 
the data. We were struck by the open resentment of community people to the 
idea of being studied by "outsiders" and were becoming increasingly sensitive 
to their legitimate concerns about what was in it for them. The staff then 
hit upon the notion of hiring and training indigenous mothers and housewives 
within the study communities to serve as testers for the three-and-a-half- 
year old children. Although training people with this background to adminis- 
ter the assessment batteries took almost twice as long as the three or four 
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weeks originally anticipated, the approach did turn out to be a viable one. 

l 

'These indigenous community people displayed an impressive degree of intrinsic / 
interest and motivation in learning the tasks, and this experience in train- 
ing and testing has produced some remarkable positive changes in their self- ^ 
concepts and aspirations. 

The Longitudinal Study of Disadvantaged Children is both an evaluation 
study of preschool programs and a basic research study of child development. 

It illustrates the power and the promise of evaluative research. It shows 
that, properly carried out with appropriate assessment of the multiple inter- 
acting factors that influence learning, the evaluation of educational pro- 
grams ^ research on educational process. 

It is therefore recommended that Section 10 of the Comprehensive Preschool 
Education and Child Day-Care Act be amended to include specific provisions for 
research and evaluation not only as groundwork for, or as an adjunct to, but as 

■ an integral on-going part of the programs of services proposed. This might be 
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accomplished by requiring that some of the funds for individual programs be 
reserved for purposes of evaluation, as was done in some of the titles of the 
Elementary and Secondary Education Act. This would make it possible to under- 
• take evaluative research as an integral pai’t of on-going programs, to gather - 
information useful for purposes of program improvement and understanding as 
well as for purposes of accountability. Since evaluative research to some ex- 
tent represents a bridge between basic research and prt5^am developmeTitr7~ort.^ 
not only offers a mechanism for continuous program renewal and improvement, 
but also an important opportunity to advance-social science'^as well as the 



social welfare. 
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